Systems and methods for highly parallel processing of parameterized simulations

ABSTRACT

Systems and associated methods for highly parallel processing of parameterized simulations are described. Embodiments permit processing of stochastic data-intensive simulations in a highly parallel fashion in order to distribute the intensive workload. Embodiments utilize methods of seeding records in a database with a source of pseudo-random numbers, such as a compressed seed for a pseudo-random number generator, such that seeded records may be processed independently in a highly parallel fashion. Thus, embodiments provide systems and associated methods facilitating quicker data-intensive simulation by enabling highly parallel asynchronous simulations.

BACKGROUND

The subject matter described herein generally relates to parallelprocessing of a plurality of stochastic simulations, wherein thebehavior of each simulation depends upon a set of input parametervalues.

Parallel computing involves executing many calculations simultaneouslyusing the principle that large problems can be divided into smaller onesand solved concurrently. One or more computing devices (computers,servers, et cetera) are often employed to carry out parallelcomputations. Various hardware arrangements support parallelism, forexample a multi-core or multi-processor computer having multipleprocessing elements within a single machine, or clusters of multiplecomputers arranged to participate in executing parallel computing tasks.The well known Hadoop software system, for example, allows robustparallel processing over a cluster of thousands of commodity machines.

Stochastic simulation is a numerical technique for predicting orstudying the behavior of systems or phenomena that are characterized byrandomness. For such systems, the “performance measures” of interest,such as the future price of a stock option or the predicted insulinlevel of a patient after taking a prescription drug, are described by aprobability distribution over the possible values of the performancemeasure. Simulation is used when this probability distribution is toocomplex to be computed analytically, and proceeds by repeatedlygenerating pseudo-random samples from the probability distribution ofinterest.

The process of generating an individual sample is called a Monte Carloreplication, and may involve a complicated series of computations. TheMonte Carlo replications are used to estimate important characteristicsof the probability distribution. For example, the expected value of theprobability distribution can be estimated by the average of the MonteCarlo samples. In practice, pseudo-random numbers are used to emulatethe randomness of a real-world system on a computer. A stream (sequence)of pseudo-random integers, called seeds, is generated according to adeterministic algorithm, but the algorithm is designed so that thesequence of seeds appears random to an observer. For example, the seedsequence will pass standard statistical tests for randomness. Thesequence of seeds is usually transformed via simple numerical operationsinto to a sequence of pseudo-random uniform numbers that are then usedin the simulation.

Exploiting the fact that different Monte Carlo replications can becomputed independently of each other, traditional parallel simulationtechniques attempt to speed up simulation processing by executing theMonte Carlo replications for the system in parallel, and sometimes alsoby decomposing the system to be simulated into disjoint components thatcan be simulated in parallel. In this approach, the degree ofparallelism is equal to the number of available processors, which canrange from tens to thousands. In general, the behavior of a simulationdepends on a plurality of input parameters; in the traditional parallelsetting, a set of parameters is associated with each parallel processor.The amount of “seeding” information used to provide a stream ofpseudo-random numbers at a processor may be large.

BRIEF SUMMARY

Embodiments broadly contemplate systems and associated methods forhighly parallel processing of parameterized simulations. Embodimentspermit processing of stochastic data-intensive simulations in a highlyparallel fashion in order to distribute the intensive workload.Embodiments utilize methods of seeding records in a database with asource of pseudo-random numbers, such as a compressed seed for apseudo-random number generator, such that seeded records may beprocessed independently in a highly parallel fashion. Thus, embodimentsprovide systems and associated methods facilitating quickerdata-intensive simulation by enabling highly parallel asynchronoussimulations. Because the simulation parameters are associated with arecord instead of a fixed processor, as is done traditionally, andbecause the source of pseudo-random numbers needed for the simulationassociated with a record is stored in a compact manner within the recorditself, the simulation workload can be distributed among processors in aflexible and dynamic manner. Besides being more efficient and allowingbetter load balancing when compared to conventional techniques, overallprocessing is more robust (for example, handling failure of processingnodes) since the records (for example, at a failed node) can be movedto, or duplicated at, a different, functioning node.

In summary, one aspect provides a method comprising: assigning recordsof a database for seeding; seeding each record, said seeding furthercomprising adding a unique source of pseudo-random numbers to eachrecord; and providing seeded records to a plurality of processing nodesfor parallel asynchronous simulations.

Another aspect provides a computer program product comprising: acomputer readable storage medium having computer readable program codeembodied therewith, the computer readable program code comprising:computer readable program code configured to assign records of adatabase for seeding; computer readable program code configured to seedeach record, wherein to seed each record further comprises adding aunique source of pseudo-random numbers to each record; and computerreadable program code configured to provide seeded records to aplurality of processing nodes for parallel asynchronous simulations.

A further aspect provides a system comprising: one or more processors;and a memory operatively connected to the one or more processors;wherein, responsive to execution of computer readable program codeaccessible to the one or more processors, the one or more processors areconfigured to: assign records of a database for seeding; seed eachrecord, wherein to seed each record further comprises adding a uniquesource of pseudo-random numbers to each record; and provide seededrecords to a plurality of processing nodes for parallel asynchronoussimulations.

The foregoing is a summary and thus may contain simplifications,generalizations, and omissions of detail; consequently, those skilled inthe art will appreciate that the summary is illustrative only and is notintended to be in any way limiting. For a better understanding of theembodiments, together with other and further features and advantagesthereof, reference is made to the following description, taken inconjunction with the accompanying drawings. The scope of the inventionwill be pointed out in the appended claims.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 illustrates an example pseudo-random number generator cycle ofseeds.

FIG. 2 illustrates example records and their respective pseudo-randomnumber generator cycles of seeds.

FIG. 3A illustrates an example seeding technique.

FIG. 3B illustrates an example seeding technique.

FIG. 4 illustrates example runtime results as a function of the numberof servers for an example embodiment.

FIG. 5 illustrates an example computer system.

DETAILED DESCRIPTION

It will be readily understood that the components of the embodiments, asgenerally described and illustrated in the figures herein, may bearranged and designed in a wide variety of different configurations inaddition to the described example embodiments. Thus, the following moredetailed description of the example embodiments, as represented in thefigures, is not intended to limit the scope of the claims, but is merelyrepresentative of those embodiments.

Reference throughout this specification to “embodiment(s)” (or the like)means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least oneembodiment. Thus, appearances of the phrases “according to embodiments”or “an embodiment” (or the like) in various places throughout thisspecification are not necessarily all referring to the same embodiment.

Furthermore, the described features, structures, or characteristics maybe combined in any suitable manner in one or more embodiments. In thefollowing description, numerous specific details are provided to give athorough understanding of example embodiments. One skilled in therelevant art will recognize, however, that aspects can be practicedwithout one or more of the specific details, or with other methods,components, materials, et cetera. In other instances, well-knownstructures, materials, or operations are not shown or described indetail to avoid obfuscation.

In this regard, although specific, non-limiting example use contexts areprovided herein to facilitate understanding of certain aspects of theinvention, as claimed, the embodiments are not so limited. Thus,embodiments broadly provide systems and associated methods for highlyparallel processing of parameterized simulations in addition to thechosen examples described herein. It should therefore be understood bythose having ordinary skill in the art that embodiments generally permitprocessing of data-intensive simulations in a highly parallel fashionvia seeding records, and providing these seeded records to processingnodes for asynchronous, parallel simulations, even though specificexample embodiments are described herein with respect to running aspecific type of simulation (Monte Carlo) in parallel.

As modern enterprises have begun to collect and analyze massive amountsof data for purposes of risk assessment and decision-making, the need topredict massive numbers of uncertain performance measures has led to theneed for a new kind of data-intensive simulation. In this setting,millions or billions of simulations need to be run. Each such simulationinvolves the generation of hundreds or thousands of Monte Carloreplications, and has its own set of input parameters. The inputparameters are often stored as a record in a database, so that eachrecord gives rise to an individual simulation.

As one example, one may wish to predict the expected insulin levels atsome future time point for each individual in the United States, basedon a stochastic model of insulin level change over time. One mighttherefore need to generate hundreds of Monte Carlo replications for eachindividual. The input parameters for an individual might be stored as arecord in a database and include factors such as age, weight, anddietary information. It is clearly desirable to be able to simulate thepatients in parallel.

Another example is provided by the Monte Carlo Database System (MCDB)for executing Business Intelligence (BI) queries over uncertain data ina relational database. Besides uncertainty arising from the need topredict future data values, as discussed above, data containinguncertainty is becoming more prevalent due to the increasing use of dataintegration, the increasing use of automated information extraction, themaking of data anonymous for privacy protection, as well as the growingprevalence of radio frequency identification (RFID) and sensor data.

In the MCDB system, a probability distribution is associated with theuncertain data values in the database. The user specifies variablegeneration (VG) functions, which are used to generate samples of theuncertain data values. Conceptually, an overall Monte Carlo replicationis obtained by generating a sample for each uncertain data value andthen running the BI query of interest over the resulting instance of thedatabase. This process is repeated multiple times, resulting in a set ofsamples from the probability distribution of the query result.

These samples are then used to estimate properties of the query-resultdistribution, such as the expected value or variance of the queryresult. MCDB actually proceeds by performing an independent simulationfor each record in the database that contains one or more uncertainvalues; this simulation comprises a plurality of Monte Carloreplications. The simulation for each such record is in generalparameterized by a set of values tailored to the record. Because thecomputational requirements of MCDB are extremely large, it is desirableto parallelize such computations. An extensive description of the MCDBSystem can be found at: R. Jampani, F. Xu, M. Wu, L. L. Perez, C. M.Jermaine, and P. J. Haas. MCDB: a Monte Carlo approach to managinguncertain data, ACM SIGMOD, pages 687-700, 2008, incorporated byreference here.

The inventors have recognized that data-intensive simulations, whichinclude a large amount of processing workload, can benefit fromparallelization. However, existing systems and methods for executingparallel simulations are not currently capable of flexiblyparameterizing simulations or achieving a sufficiently high degree ofparallelism. Accordingly, embodiments are described herein that flexiblyprovide a high degree of parallelism for processing data-intensivesimulations.

Embodiments employ methods for highly parallel processing ofparameterized simulations. Each simulation can be parameterizedaccording to the values in a corresponding database record. The databasemay be a relational database, a flat file of data, a collection of datain Javascript Object Notation (JSON) format, a multidimensional array ofdata values, and so forth. An example method for parallel execution ofmultiple simulations, as described further herein includes: assigningeach record to exactly one of n parallel-processing nodes for seeding;seeding the records, where the nodes seed their assigned recordsasynchronously and in parallel (where seeding a record may amount toadding one or more compressed pseudo-random number seeds to the record);assigning each seeded record to exactly one of m parallel-processingnodes for simulation; and executing one simulation per seeded record,asynchronously and in parallel at each node, where the nodes expand thecompressed pseudo-random number seed prior to simulation execution.

An alternative embodiment assigns each record to a plurality of nodes,with each node executing a subset of the total (Monte Carlo)replications for the simulation associated with the seeded record. Oncethe simulations are run, the results can be recorded and queries,visualizations, or other processing can be executed over the results.

The generation of pseudo-random numbers is a component of simulationsdescribed herein. The pseudo-random numbers provide (statistically)random values for use in the simulations. Accordingly, a brief overviewof pseudo-random number generators (PRNG) follows.

Usually, a PRNG is initiated with a starting seed s₀, and then generatesa sequence of seeds s₁, s₂, . . . by using a deterministic recursion ofthe form s_(i+1)=T₁(s_(i)). At each step, the generator typically uses asecond transformation to create a (typically 32-bit) integerr_(i)=T₂(s_(i))∈{1, 2, . . . , 2³²−1}, which is further transformed to apseudo-random uniform number u_(i) on [0, 1] via normalization:u_(i)=r_(i)/2³². The transformations T₁ and T₂ depend on the specificgenerator, as do the number of bits in each s_(i) (typically a multipleof 32). For a good generator, the sequence u₁, u₂, . . . isstatistically indistinguishable from a sequence of “true” i.i.d. samplesfrom the uniform [0, 1] distribution, in the sense that the sequencewill pass statistical tests for uniformity and randomness. The uniformpseudo-random numbers can then be transformed into pseudo-random numbershaving a user-specified distribution.

As illustrated in FIG. 1, the sequence of seeds s₀, s₁, . . . , s_(n−1)produced by a PRNG eventually loops back on itself; that is, s_(N)=s_(O)for some number N with s_(i)≠s_(j) for 0≤i<j<N, thus forming a cycle ofseeds. The number N is called the cycle length of the PRNG.

When a simulation is executed for a given record, the simulationprocedure is “fed” an initial seed s_(i) that was originally attached tothe record during the seeding phase. The simulation procedure then callsa PRNG multiple times to consume a sequence of k seeds s_(i), s_(i+1), .. . , s_(i−k) during the course of the simulation. Depending on thesimulation routine, the number k of seeds consumed might not be easilypredictable in advance. For example, functions that use“acceptance-rejection” methods to generate non-uniform random variatesuse a “random” number of seeds.

As illustrated in FIG. 2, it is necessary that the seed sequences (301a, 301 b, . . . 301 n) for the records (Record 1, Record 2, . . . Recordn) be mutually disjoint, since overlaps cause unintended statisticaldependencies between records (Record 1, Record 2, . . . Record n), whichcan lead to incorrect simulation results. Since the number of recordscan run to the billions, and each record can consume millions of seeds,the cycle length of the underlying PRNG must be very long. Fortunately,state-of-the-art PRNGs can achieve cycle lengths of 2⁵⁰⁰ and longer,while retaining good statistical properties.

A problem in terms of parallelization, however, is that these long cyclelengths are achieved by using very long seeds, which can greatly inflatethe size of the records. Such inflation can significantly slow downprocessing by inducing large processing overheads when moving the dataaround for parallelization. Still further, for parallelization as perthe example embodiments discussed herein, even more random numberstreams are needed than the number of records. For example, using theintra-record parallelism scheme (described further herein), both seedingand subsequent stream generation must be done in a highly parallelizablemanner. A substantial literature has developed around the problem ofgenerating pseudo-random number streams in parallel, largely motivatedby scientific applications. Unfortunately, none of the previouslyproposed methods is directly applicable for a variety of reasons.

For example, in many prior algorithms the number of streams neededcoincides with the number of processors, and hence is on the order of10¹ to 10³. However, parallelization on the order contemplated byembodiments herein requires on the order of 10⁶ to 10⁹ streams.“Leapfrog” methods (in which successive seeds from a base generator aredealt out to the streams like playing cards from a deck) are simple, butcan suffer from statistical anomalies. A number of methods use a simplelinear congruential generator as the base generator, which yields aninadequate cycle length.

The most promising parallel PRNGs appear to be the SPRNG generator andthe PLFG generator, each of which uses a lagged Fibonacci generator asthe base generator and exploits special properties of this class ofgenerators. A key drawback of such generators is that the seedcorresponds to a “lag table” that typically comprises tens of thousandsof 32 bit integers and will not fit into a record and thus cannot beused for parallelization of data-intensive simulations.

Accordingly, example embodiments take the classical “cycle splitting”approach using huge-period generators, and limit the amount of seedinginformation in a record by employing techniques for combining generatorsand for skipping ahead on a cycle. Thus, for implementing theparallelization schemes, as per the example embodiments discussedherein, specialized techniques are utilized to provide adequate seeding.

Example embodiments are described herein with respect to two simulationparallelization schemes that are applicable to simulation workloads:inter-record and intra-record parallelism. These schemes are nowdescribed. Distributed seeding techniques allowing such parallelizationschemes to be employed are discussed thereafter.

Inter-record parallelism is obtained by simulating multiple records inparallel. For inter-record parallelism, a single processing nodegenerates all of the Monte Carlo replications for a given record.Inter-record parallelism is naturally supported, for example, byHadoop's partitioned parallelism mechanism. When there are fewer recordsthan nodes, however, some nodes will be idle. This problem becomesparticularly acute when the Monte Carlo replications are computationallyexpensive (such as in many financial simulations) or many replicationsare needed per record. In such cases, use of intra-record parallelismallows the Monte Carlo replications for a single record to be performedby multiple nodes, as described next.

For intra-record parallelism, the Monte Carlo replications for a recordare conceptually partitioned into disjoint chunks. For δ chunks, therecord is duplicated δ times, and each duplicate is then also called achunk. A chunk ID and seed are stored with each chunk, and thesimulation based on the chunk only generates those Monte Carloreplicates associated with the chunk. Thus, after splitting the recordinto chunks, different chunks, and hence different subsets of the MonteCarlo replications for the original record, can be generatedasynchronously and in parallel on different nodes, allowing better loadbalancing. In other words, the inter-record parallelism processingmethod is applied, but on the multiple chunks rather than on theoriginal records. By using this latter approach, all of the nodes in thecluster can be fully utilized at all times.

The number of chunks per record depends on the available systemresources. Outside of some minor modifications, seeding and simulationproceed in the same manner as with the inter-record parallelismapproach. After the multiple chunks of a single record have all beensimulated, the resulting Monte Carlo replications can be mergedtogether. Using these example parallelism schemes, both seeding andsubsequent pseudo-random number stream generation must be done in ahighly parallelizable manner. As no previously developed techniquepermits a sufficiently high degree of parallelism, flexibility in loadbalancing, and robustness to node failures, distributed seedingtechniques utilized by example embodiments are now described.

Example embodiments utilize new methods for seeding records and forsubsequently using the seeding information for purposes of record-basedsimulation in connection with highly parallel Monte Carlo computations.However, it should be understood that the seeding techniques describedherein are equally applicable to other parallel simulation contexts.

A first method, referred to herein as SeedSkip, is based on recentskip-ahead methods for huge-period PNRGs. SeedSkip guarantees that therewill be no collisions between streams of PRNG seeds consumed bydifferent records, that is, that the streams will be mutually disjoint,and requires only a small amount of seeding information per record,making it suitable for highly parallelized computations. Further,seeding can proceed asynchronously and in parallel on multiple nodes.

Because the skip-ahead functionality is rather complex and has not yetbeen widely adopted, and because the method requires knowledge aboutsimulation-procedure properties that may not be readily available, alsoprovided is a fall-back method. The fall-back method is referred toherein as SeedMult, uses multiple PRNGs, and requires neither skip-aheadmethods nor detailed knowledge about the internals of the simulationprocedure. The SeedMult method does not completely rule out thepossibility of collisions between streams, but it is shown herein thatthe probability of such collisions is vanishingly small. As withSeedSkip, seeding can proceed asynchronously and in parallel on multiplenodes.

For ease of description, it is assumed that the simulation associatedwith each record needs to consume only a single stream of PRNG seedsduring simulation. It is also assumed that there are exactly J (≥1)nodes available to execute, in parallel, the seeding operation. Thesenodes are denoted by P₀, P₁, . . . , P_(J−1). For ease of description,it is initially assumed that the inter-record parallel processing schemeis used for the simulation step, so that a single node generates all ofthe Monte-Carlo realizations associated with a record. Under theforegoing assumptions, each record needs to be seeded with exactly oneseed.

To more readily appreciate certain aspects of the SeedSkip method, abrief overview of skip-ahead techniques is first given, followed by adescription of methods exploiting some characteristics of thesetechniques for seeding. Regarding skip-ahead techniques for PRNGs, aPRNG with seed cycle s₀, s₁, . . . , s_(N−1) supports skipping-ahead iffor one or more values of v>1, it implements an efficient transformationT^((v)) such that T^((v))(s_(i))=s_(i+v) for all i. That is, thegenerator can skip v positions forward on the cycle without having togenerate the intermediate seeds s_(i+1), s_(i+2), . . . , s_(i+v−1).Typically, one must predetermine a desired skip length v and thenpre-compute a set of generator-dependent “skip parameters” that supportthe desired skip length. This pre-computation is usually expensive andcan be quite complicated for modern huge-period generators but, once theparameters are computed, skips of length v can be performed veryefficiently, so that the pre-computation cost is amortized over manyskips.

For example, many popular types of PRNGs belong to the family of“combined multiple recursive generators” (CMRGs). An individual multiplerecursive generator (MRG) has a linear transformation function T.Specifically, for a seed s that is a vector of k 32-bit integers, an MRGrecursively sets s_(n)=As_(n−1) mod m for a k×k matrix A and largeinteger m. A CMRG maintains k MRGs in parallel, where typically k≤4, andproduces the overall random integer r_(n) at the n^(th) step bycombining the k seeds s_(n) ¹, s_(n) ², . . . , s_(n) ^(k), for example,by setting r_(n)=(s_(n) ¹+s_(n) ²+ . . . +s_(n) ^(k)) mod m′ for anappropriate value of m′. An efficient technique for skipping ahead by afixed number of steps v in a CMRG has been developed. The idea is topre-compute the matrix A^(v) mod m using a “divide and conquer” scheme,and then exploit the fact that, for an MRG, s_(n+v)=(A^(v)s_(n)) modm=(A^(v) mod m)(s_(n)) mod m.

As another example, consider the recent WELL512 generator. WELL512 is ahuge-period generator having a period of 2⁵¹² and improves on thestatistical properties of the Mersenne Twister, a well known huge-periodPRNG. WELL512 is also based on a linear recurrence, but with the modulusequal to 2, so that bits are manipulated individually. Efficientskip-ahead techniques for such generators have very recently beenproposed. These techniques are nontrivial to implement, involving therepresentation of the linear transformation s

As as a formal series, appropriately manipulating the seriescoefficients to represent the skip, and then effecting the actual skipsusing the modified series. In this case, for a given skip length, onepre-computes and stores the modified coefficients that correspond to theskip.

FIG. 3A illustrates an example SeedSkip skip-ahead method employed byexample embodiments. The SeedSkip method employed by example embodimentsuses a single PRNG for both seeding and simulation. The single PRNG(denoted by G and using WELL512 for an example implementation) requiresa known upper bound k* for the number of records seeded per node.Moreover, the single PRNG requires a known upper bound of ρ* for thenumber of G seeds used up during any Monte Carlo replication. Recallthat each record invokes the simulation procedure ω times, where ω isthe number of Monte Carlo replications.

In SeedSkip, a node trivially “seeds” 310A a record with its node numberand intra-node record identifier. These quantities are referred toherein as n_(p) and n_(i), and note that n_(p)∈[0,J−1] andn_(i)∈[0,k*−1]. To naively simulate 320A a record, n_(p)k*+n_(i) skips330 of length γ=ωρ* are then generated, which provides the startingpoint 340A on the G cycle for the simulation. To actually execute thesimulation, at most γ seeds for the record are used. In fact, the numberof required skips can be vastly reduced by storing a carefully selectedset of skip parameters, using O(log₂(N/γ)) storage.

Specifically, suppose that N/γ=2^(j) for some integer j. Skip parametersare computed that correspond to skip lengths v=γ, 2γ, . . . , 2^(J−1)γ.Then O(log₂(n_(p)k*+n_(i))) skips along the G cycle are required foreach record to get to the starting point for the record's simulation.That is, denoting by b₁b₂ . . . b_(j) the binary representation ofn_(p)k*+n_(i), execute a skip of length 2^(b) ¹ γ if and only if b₁=1,then execute a skip of length 2^(b) ² γ if and only if b₂=1, and soforth.

The SeedSkip method cannot be used if a good implementation of theadvanced skip-ahead technology for huge-period generators is not readilyavailable or, more importantly, if the simulation procedures are socomplicated that it is impossible to obtain a reasonable upper bound onthe number of seeds consumed during a Monte Carlo replication. In thiscase, the SeedMult seeding method can be used instead.

FIG. 3B illustrates an example SeedMult method employed by exampleembodiments. In the following, k_(j) denotes the number of records thatnode P_(j) is assigned to seed, and l_(i) denotes the number ofsuccessive seeds consumed by record t_(i) is during a Monte Carloreplication.

The SeedMult method uses four PRNGs, denoted G¹-G⁴; generators G¹ and G²are used for seeding, and generators G³ and G⁴ are used for simulation;it is assumed that G⁴ is a huge-period generator. A seed s_(j) ^(i) forgenerator G^(i) (i=1, 2, 3, 4) is an array (s_(j,1) ^(i), s_(j,2) ^(i),. . . , s_(j,m) _(i) ^(i)) of m_(i) 32-bit integers. The j^(th)execution of G¹ updates the current seed s_(j−1) ^(i) via atransformation s_(j) ^(i)=T₁ ^(i)(s_(j−1) ^(i)) and, using a secondtransformation, creates a 32-bit integer r_(j) ^(i)=T₂ ^(i)(s_(j)^(i))∈{1, 2, . . . , 2³²−1}, which is normalized to a pseudo-randomuniform number u_(j) on [0, 1] via u_(j) ^(i)=r_(j) ^(i)/2³². PRNG G^(i)has cycle length N_(i).

Each node P_(j)(0≤j≤J−1) executes the following procedure for seeding320B its local records:

-   -   Initialize generator G¹ with a fixed seed s₀ ¹. All nodes use        the same seed.    -   Generate m₂(j+1) successive random integers r₀ ¹, R₁ ¹, . . . ,        r_(m) ₂ _((j+1)−1) ¹ from G¹, and use the last m₂ of these        integers to form an initial seed s₀ ²(j) to use with G². That        is, s₀ ²(j)=(r_(jm) ₂ ¹, r_(jm) ₂ ₊₁ ¹, . . . , r_((j+1)m) ₂ ⁻¹        ¹).    -   Use G² to generate a sequence of k_(j)m₃ random integers r₀        ²(j), r₁ ²(j), . . . , r_(k) _(j) _(m) ₃ ⁻¹ ²(j), and use the        i^(th) subsequence of length m₃ to seed the i^(th) local record.        More specifically, setting K₀=0 and K_(l)=k₁+k₂+ . . . +k_(j)        for l≥1, seed record t_(K) _(j) _(+i) with so s₀        ³(K_(j)+i)=(r_(im) ₃ ²(j), r_(im) ₃ ₊₁ ²(j), . . . , r_((i+1)m)        ₃ ⁻¹ ²(j)) for 0≤J−1 and 0≤i≤k_(i)−1.

Later, at simulation time (which may be performed by the same node asused for seeding or a different node), the following procedure is usedto simulate 320B record t_(i), which has previously been seeded withseed s₀ ³(i) as computed above.

-   -   Initialize G³ with seed s₀ ³(i), and use G³ to generate m₄        successive random integers to form a seed s₀ ⁴(i).    -   Initialize G⁴ with s₀ ⁴(i), and use G⁴ to generate the l_(i)        random numbers needed by simulation associated with t_(i).

The following result supports analysis of the SeedMult algorithm.

LEMMA 1. Consider a PRNG having a cycle of length N comprising seeds s₁,s₂, . . . , s_(N) and having seed transformation function T. Fix K>1 andlet M₁, M₂, . . . , M_(K) be K mutually independent random variables,each uniformly distributed on {1, 2, . . . , N}. For k=1, 2, . . . , K,define a segment σ_(k) of length L(≥1) by setting σ_(k,1)=s_(M) _(k) andσ_(k,l)=T(σ_(k,l−1)) for 2≤l≤L. Then the probability that the K segmentshave one or more overlaps is less than 2K²L/N.

The probabilities of overlaps during the seeding and simulations can nowbe analyzed. For the seeding process, observe that, by construction, theinitial G²-seeds s₀ ²(0), s₀ ²(1), . . . , s₀ ²(J−1) for the J nodes arebased on mutually disjoint segments of the G¹ cycle. These J seeds forG², can be viewed as being randomly placed, uniformly and independently,on the cycle for G². Each seed s₀ ²(j) initiates a segment of k_(j)m₃successive seeds on the G² cycle. With k*=max_(0≤j≤J−1)k₁, Lemma 1implies that the probability of any overlaps between these segments isless than α₂=2J²k*m₃/N₂. For the simulation process, set K=k₀+k₁+ . . .+k_(j−1) and observe that the K seeds s₀ ³(0), s₀ ³(1), . . . s₀ ³(K−1)can be viewed as being randomly placed on the cycle for G³, and theprobability of any overlaps between segments (each having length m₄) isless than α₃=2K²m₄/N₃. Finally, let l*=max_(1≤i≤K)l_(i) be the maximumnumber of seeds consumed by a record during a simulation, and view the Kseeds S₀ ⁴(0), s₀ ⁴(1), . . . s₀ ⁴(K−1) as being randomly placed on thecycle for G⁴. Since each seed initiates a segment of length at most l*,the overlap probability is less than α₄=2K²l*/N₄.

With suitable choice of generators, m₃ can be chosen to be a smallinteger, so that seeding a record does not unduly increase the recordsize. Moreover, α₂, α₃, and α₄ can be made vanishingly small, so that,effectively with probability 1, no seed is ever used more than once forany of the generators. Specifically, the generators in Table 1 can beused. When J=2¹⁰≈1,000, k_(i)=2²⁰≈1,000,000 for 1≤i≤J (so that there areK=2³⁰, or about 1 billion records in total), and l_(i)=2²⁰ for 1≤i≤K,then, for the above choice of generators, we have α₂≈2⁻¹⁴⁸≈10⁻⁴⁴,α₃≈2⁻⁵⁶≈10⁻¹⁷, and α₄≈2⁻⁴³¹≈10⁻¹²⁹. Clearly, the probability of anyoverlaps is negligible, and only four 32-bit integers of seedinginformation need be stored in each record. The selected generators,besides having the requisite seed sizes and cycle lengths, are alsoknown to have good statistical properties.

TABLE 1 Generators for seeding and simulation. Seed length Cycle Lengthi Name m_(i) N_(i) 1 LCG16807 1 ≈2³¹  2 MRB32ka 6 ≈2¹⁹¹ 3 CLCG4 4 ≈2¹²¹4 WELL512a 16 ≈2⁵¹²

Some of the simplifying assumptions made herein can be dropped and theprocesses described above can be extended in several important ways.

For intra-record parallelism, the scheme uses multiple nodes to generatethe N Monte Carlo replications for a record during simulation. That is,the set of Monte Carlo replications is divided into δ(>1) chunks ofω=N/δ replications each, and these chunks are generated in parallel (bya set of J′ nodes). Inter-record parallelism corresponds to the specialcase δ=1. The SeedSkip and SeedMult processes are modified to assigneach chunk its own compressed seed. The previous descriptions andanalyses still hold, except that now each k_(j) is interpreted as thenumber of chunks that are seeded by node P_(j), and ω is interpreted, asabove, to be the number of Monte Carlo replications per chunk.

For multiple VG functions, suppose that we need M(>1) compressed seedsper record to perform a simulation. For example, it may be convenient touse M streams of pseudo-random numbers to represent M independent randomphenomena that occur within the simulation. As with intra-recordparallelism, the prior descriptions and analyses carry over virtuallyunchanged, except that now each (stream, chunk) pair is assigned aunique seed. With a slight abuse of terminology, henceforth the term“chunk” is used to denote such a pair (thus, there are Mδ chunks intotal for each record, and k_(j) is again interpreted as the number ofchunks that are seeded by P_(j)).

In terms of shared storage, if the J nodes have access to a (small)amount of shared storage, as is the case for Hadoop, then speedups forSeedMult are possible by amortizing seeding costs over multipledata-intensive simulation experiments. For example, suppose that thenumber of seeding nodes used per experiment is bounded above by J* overthe experimental workload, and the upper bound k* on the number ofchunks per node also holds for the workload. Then the seeds s₀ ²(0), s₀²(1), . . . , s₀ ²(J*−1) can be computed once and written to sharedstorage. These seed values can simply be looked up at seeding time (inparallel, since only read operations are involved). Similarly, theSeedSkip scheme can also exploit shared storage to amortize the initialcost of computing the skip-ahead information over an experimentalworkload. For example, if the values J=2¹⁰≈1,000, k_(i)=2²⁰≈1,000,000serve as upper bounds for the workload, then 30 sets of skip parametersare required, and these can be pre-computed and stored for use by all ofthe experiments.

One skilled in the art can see that hybrids of SeedMult and SeedSkip canbe developed by using skip-ahead methods to place starting seeds on theG² and/or G³ cycles in an equally-spaced manner, rather than droppingthem randomly. Similarly, variants of SeedSkip that use fewer than fourPRNGs can be developed.

As an example of scaleup for inter-record parallelism, the running timesof several MCDB-style queries on increasing data sets were measuredwhile proportionally increasing the number of servers. In this setting,each tuple is simulated using the methods described above, and then adatabase query is run over the results of the simulation. In the idealcase, it is expected that the running time would remain the same as thedata set and number of servers are proportionally increased.

In the example case, the well known TPC-H benchmark dataset was used.The dataset was generated using the dbgen program provided by thebenchmark, converted to JSON, and loaded into Hadoop's HDFS (distributedfile system). Query Q1 and Q4 from Jampani et al. supra, were used (bothof which are described in detail therein). Query Q1 guesses the revenuegain for products supplied by companies in a next year assuming thatcurrent sale trends hold. Query Q4 estimates the effect of a 5% customerprice increase on an organization's profits.

For Queries Q1 and Q4, the number of servers was varied from 1 to 10,and 10 different datasets were prepared (from 2 GB for 1 server to 20 GBfor 10 servers). For each computing-cluster size, both Q1 and Q4 wereevaluated and total elapsed time was recorded. Each query was evaluatedusing both the SeedMult and SeedSkip distributed seeding methods, and1000 Monte Carlo replicates were produced.

FIG. 4 illustrates the results for Q1 and Q4 using SeedSkip. Theillustrated execution times are the average of several runs. For bothqueries, the time for seeding was insignificant compared to the time forI/O, sorting, network transfer, and simulation. As a result, significantdifferences in overall elapsed times when SeedSkip or SeedMult were usedwere not observed. In isolation, it appeared that SeedSkip was at least5% faster than SeedMult. For both Q1 and Q4, join operations accountedfor nearly 50% of the total execution time. Q1 spent most of theremaining time on record manipulation (for example, aggregating arrays)rather than on simulation. In comparison, Q4 's simulation procedure wasmore complex, so substantially more time was spent on simulation than onrecord manipulation.

The network was not the bottleneck for either query. Both queries spenta large fraction of their time reading, writing, sorting, andtransferring data. Q4 also spent substantial time on its CPU-intensiveVG function. The curves in FIG. 4 are relatively flat, which indicatesgood scale-up.

Referring to FIG. 5, it will be readily understood that embodiments canbe implemented using any of a wide variety of devices or combinations ofdevices, such as the servers (computers) described herein. An exampledevice that may be used in implementing one or more embodiments includesa computing device in the form of a computer 510. In this regard, thecomputer 510 may execute program instructions configured to performdistributed seeding and highly parallelized data-intensive simulations,and perform other functionality of the embodiments, as described herein.

Components of computer 510 may include, but are not limited to,processing units 520, a system memory 530, and a system bus 522 thatcouples various system components including the system memory 530 to theprocessing unit 520. Computer 510 may include or have access to avariety of computer readable media. The system memory 530 may includecomputer readable storage media in the form of volatile and/ornonvolatile memory such as read only memory (ROM) and/or random accessmemory (RAM). By way of example, and not limitation, system memory 530may also include an operating system, application programs, otherprogram modules, and program data.

A user can interface with (for example, enter commands and information)the computer 510 through input devices 540. A monitor or other type ofdevice can also be connected to the system bus 522 via an interface,such as an output interface 550. In addition to a monitor, computers mayalso include other peripheral output devices. The computer 510 mayoperate in a networked or distributed environment using logicalconnections to one or more other remote computers or databases, such asa cluster of machines for parallel computations. The logical connectionsmay include a network, such local area network (LAN) or a wide areanetwork (WAN), but may also include other networks/buses.

It should be noted as well that certain embodiments may be implementedas a system, method or computer program product. Accordingly, aspects ofthe invention may take the form of an entirely hardware embodiment, anentirely software embodiment (including firmware, resident software,micro-code, et cetera) or an embodiment combining software and hardwareaspects that may all generally be referred to herein as a “circuit,”“module” or “system.” Furthermore, aspects of the invention may take theform of a computer program product embodied in one or more computerreadable medium(s) having computer readable program code embodiedtherewith.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, et cetera, or any suitablecombination of the foregoing.

Computer program code for carrying out operations for aspects of theinvention may be written in any combination of one or more programminglanguages, including an object oriented programming language such asJava™, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer (device), partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider).

Aspects of the invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatuses(systems) and computer program products according to exampleembodiments. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

This disclosure has been presented for purposes of illustration anddescription but is not intended to be exhaustive or limiting. Manymodifications and variations will be apparent to those of ordinary skillin the art. The example embodiments were chosen and described in orderto explain principles and practical application, and to enable others ofordinary skill in the art to understand the disclosure for variousembodiments with various modifications as are suited to the particularuse contemplated.

Although illustrated example embodiments have been described herein withreference to the accompanying drawings, it is to be understood thatembodiments are not limited to those precise example embodiments, andthat various other changes and modifications may be affected therein byone skilled in the art without departing from the scope or spirit of thedisclosure.

What is claimed is:
 1. A non-transitory computer program productcomprising: a non-transitory computer readable storage medium havingcomputer readable program code embodied therewith, the computer readableprogram code comprising: computer readable program code configured toassign records of a database for seeding to one of a plurality ofparallel-processing nodes, wherein each of the records is assigned to asingle of the plurality of parallel-processing nodes; computer readableprogram code configured to, at each of and in parallel with theplurality of parallel-processing nodes, seed each record using the nodeassigned to the record, wherein to seed each record further comprisesadding a unique source of pseudo-random numbers to each record, whereinthe unique source of pseudo-random numbers is stored within acorresponding record; and computer readable program code configured toprovide, using a cycle of seeds, seeded records to a plurality ofprocessing nodes for parallel asynchronous simulations, wherein theplurality of processing nodes is different from the plurality ofparallel-processing nodes, wherein each of the seeded records isprovided to a single of the plurality of processing nodes, wherein theproviding comprises: with respect to the cycle of seeds, generatingskips of varied length based on the records of the database; andcomputer readable program code configured to modify, based on thegenerated skips of varied length, a start point for a cycle of at leastone simulation; and each of the seeded records undergoing a singlesimulation, wherein the pseudo-random number of the corresponding recordis utilized during the simulation.
 2. The computer program productaccording to claim 1, wherein to assign records of a database forseeding further comprises assigning records to a plurality of processingnodes for seeding independently and asynchronously.
 3. The computerprogram product according to claim 1, wherein the unique source ofpseudo-random numbers further comprises a compressed seed for apseudo-random number generator.
 4. The computer program productaccording to claim 1, wherein the parallel asynchronous simulationsfurther comprise, a plurality of simulations, each executed inaccordance with a given seeded record by at least one of the pluralityof processing nodes.
 5. The computer program product according to claim1, Wherein to add a unique source of pseudo-random numbers to eachrecord further comprises adding a source of pseudo-random numbers toeach record enabling each of the plurality of processing nodes togenerate pseudo-random numbers locally for a simulation.
 6. The computerprogram product according to claim 1, wherein to add a unique source ofpseudo-random numbers to each record further comprises providing acompressed seed to each record enabling each of the plurality ofprocessing nodes to generate a disjoint sequence of pseudo-randomnumbers for the record using a skip-ahead technique for a singlepseudo-random number generator locally.
 7. The computer program productaccording to claim 6, wherein using a skip-ahead technique furthercomprises pre-computation of a set of skip parameters whose size is alogarithmic function of the cycle length of the pseudo-random numbergenerator.
 8. The computer program product according to claim 1, whereinto provide a unique source of pseudo-random numbers to each recordfurther comprises providing a compressed seed to each record enablingeach of the plurality of processing nodes to locally generate a sequenceof pseudo-random numbers for the record that, above a given probability,is disjoint from a sequence for all other records, using a techniquebased on multiple pseudo-random number generators.
 9. The computerprogram product according to claim 1, further comprising computerreadable program code configured to record simulation results fromparallel asynchronous simulations executed with the plurality processingnodes using the seeded records.
 10. A system comprising: one or moreprocessors; and a memory operatively connected to the one or moreprocessors; wherein, responsive to execution of computer readableprogram code accessible to the one or more processors, the one or moreprocessors are configured to: assign records of a database for seedingto one of a plurality of parallel-processing nodes, wherein each of therecords is assigned to a single of the plurality of parallel-processingnodes; at each of and in parallel with the plurality ofparallel-processing nodes, seed each record using the node assigned tothe record, wherein to seed each record further comprises adding aunique source of pseudo-random numbers to each record, wherein theunique source of pseudo-random numbers is stored within a correspondingrecord; and provide, using a cycle of seeds, seeded records to aplurality of processing nodes for parallel asynchronous simulations,wherein the plurality of processing nodes is different from theplurality of parallel-processing nodes, wherein each of the seededrecords is provided to a single of the plurality of processing nodes,wherein the providing comprises: with respect to the cycle of seeds,generating skips of varied length; and modifying, based on the generatedskips of varied length, a start point for a cycle of at least onesimulation; and each of the seeded records undergoing a singlesimulation, wherein the pseudo-random number of the corresponding recordis utilized during the simulation.
 11. The system according to claim 10,wherein to assign records of a database for seeding further comprisesassigning records to a plurality of processing nodes for seedingindependently and asynchronously.
 12. The computer program productaccording to claim 1, wherein a single pseudo-random number generator isemployed to provide seeded records to a plurality of processing nodesfor parallel asynchronous simulations and for performing at least onesimulation.